[SPARK-61157][CONNECT][PYTHON] Capture the call sites of the origins of Column operations without importing a missing IPython module on every call - #346

What changes were proposed in this pull request?

SPARK-61157: Capture the call sites of the origins of Column operations without importing a missing IPython module on every call.

Every Column operation of the Python client calls _with_origin, which in PySpark captures the call site of the user code with pyspark.errors.utils._capture_call_site. That function runs import IPython and import ipykernel on every call. Where ipykernel is not installed, each failing import scans sys.path again.

• origin.rs captures the call site in Rust, with the result of _capture_call_site(1): the frames outside the pyspark package, which are "file:line", and within an IPython shell, without the frames of IPython and ipykernel, "line N in cell [K]". The frames are joined by newlines, as before.
• IPython and ipykernel are read from sys.modules. When one of them is not loaded, the import is tried once per process and a failure is not retried; a module imported later by the user is still seen.
• The vendored pyspark/errors/utils.py is unchanged, and is_debugging_enabled is unchanged.
• The call sites of errors are the same as official PySpark 4.3.0: a failing select and a failing cast in a nested function, a failing division, and the IPython cell form (checked with fake IPython and ipykernel modules). The official query-context tests (test_parity_dataframe_query_context.py) pass on the drop-in, 6 of 6.

Why are the changes needed?

With ipykernel not installed, 90k Column operations (F.col("a") + 1, * 2, .alias) took 6.8 s, against 0.23 s with debugging off. spark.range(1).select(*2000 names) five times took 0.66 s, against 0.004 s with debugging off. Official PySpark pays the same per-call cost in this helper (10.7 s for the same run on that machine); in the drop-in, whose own work per Column operation is small, nearly all of the time went to the failing imports.

After this change, the same run takes 0.77 s with debugging on, and the select 0.022 s.

Does this PR introduce any user-facing change?

No. The error messages and the query contexts are the same. Column operations are faster when IPython or ipykernel is not installed.

How was this patch tested?

• Offline test_column_origins_capture_their_call_site_without_importing_per_call: 200 Column operations on the mock server import IPython and ipykernel at most once (the counting __import__ sees 800 imports without this change), and the request of a Column built on the test line carries the string that _capture_call_site(1) gives on that line.
• The official test_parity_dataframe_query_context.py on the drop-in against a live server: 6 passed, as before.
• The tests of both crates, e2e included, against a local Spark 4.3.0 server (875 passed), the build of all tests, also with wasm-udf, the offline Python suites (2202 passed, 8 skipped), and the live drop-in, Arrow and ML suites (184 passed) and live cases (10 passed). rustfmt passes, and clippy reports no diagnostics in the changed lines.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.